Image sensing using solid-state image sensing elements

ABSTRACT

This invention makes it possible to provide a technique for suppressing a decrease in resolution of a sensed image even when an image sensor on which solid-state image sensing elements with different sensitivities are arranged is used. A demosaic unit obtains a color component of a given pixel, sensed at the first sensitivity, by performing interpolation calculation using color components of pixels each of which is adjacent to the given pixel and is sensed at the first sensitivity. The demosaic unit also obtains a color component of a given pixel, sensed at the second sensitivity, by performing interpolation calculation using color components of pixels each of which is adjacent to the given pixel and is sensed at the second sensitivity.

This application is a continuation of International Application No.PCT/JP2011/000128, filed on Jan. 13, 2011, which claims the benefit ofJapanese Patent Application No. 2010-010366, filed Jan. 20, 2010, andJapanese Patent Application No. 2010-288556, filed Dec. 24, 2010. Thecontents of the aforementioned applications are incorporated herein byreference in their entirety.

TECHNICAL FIELD

The present invention relates to a single-plate HDR image sensingtechnique.

BACKGROUND ART

The dynamic range can be widened by imparting different sensitivities toadjacent pixels and synthesizing a signal of a high-sensitivity pixeland a signal of a low-sensitivity pixel. For example, PTL1 discloses acolor filter array in which all colors: light R, G, B, and W and dark r,g, b, and w are arranged on all rows and columns. Also, PTL2 discloses asensor on which RGB rows and W rows are alternately arranged.

However, in the technique disclosed in PTL1, all pixels are provided atthe same ratio, so the sampling interval of G (Green), for example, isevery two pixels. Thus, only a resolution becomes half that of a normalBayer array. Also, when G of a high-sensitivity pixel is saturated, thesampling interval of G becomes every four pixels. Thus, a resolutionbecomes one quarter of that of the Bayer array.

In the technique disclosed in PTL2, only luminance information is usedfor a low-sensitivity pixel, so color information cannot be held for ahigh-luminance portion. Also, the resolution in the vertical directionhalves.

Citation List

Patent Literature

-   PTL1: Japanese Patent Laid-Open No. 2006-253876-   PTL2: Japanese Patent Laid-Open No. 2007-258686

SUMMARY OF INVENTION Technical Problem

As described above, when pixels with different sensitivities arearranged on the same sensor, the resolution inevitably decreases. Also,when a high-sensitivity pixel is saturated, the resolution furtherdecreases.

The present invention has been made in consideration of theabove-mentioned problem, and has as its object to provide a techniquefor suppressing a decrease in resolution of a sensed image even when animage sensor on which solid-state image sensing elements with differentsensitivities are arranged is used.

Solution to Problem

In order to achieve the object of the present invention, an imagesensing apparatus according to the present invention has, for example,the following arrangement. That is, there is provided an image sensingapparatus including an image sensor on which solid-state image sensingelements which sense color components at a first sensitivity andsolid-state image sensing elements which sense color components at asecond sensitivity higher than the first sensitivity are alternately,two-dimensionally arranged, comprising a first calculation unit thatobtains a color component of a given pixel, sensed at the firstsensitivity, by performing interpolation calculation using colorcomponents of pixels each of which is adjacent to the given pixel and issensed at the first sensitivity, in a color image based on an imagesignal output from the image sensor, a second calculation unit thatobtains a color component of a given pixel, sensed at the secondsensitivity, by performing interpolation calculation using colorcomponents of pixels each of which is adjacent to the given pixel and issensed at the second sensitivity, in the color image output from theimage sensor, and a unit that outputs the color image in which colorcomponents of all pixels are determined by the first calculation unitand the second calculation unit.

Advantageous Effects of Invention

With the arrangement according to the present invention, it is possibleto suppress a decrease in resolution of a sensed image even when animage sensor on which solid-state image sensing elements with differentsensitivities are arranged is used.

Other features and advantages of the present invention will be apparentfrom the following descriptions taken in conjunction with theaccompanying drawings. Noted that the same reference characters denotethe same or similar parts throughout the figures thereof.

BRIEF DESCRIPTION OF DRAWINGS

The accompanying drawings, which are incorporated in and constitute apart of the specification, illustrate embodiments of the invention and,together with the description, serve to explain the principles of theinvention.

FIG. 1 is a block diagram illustrating an example of the functionalconfiguration of an image sensing apparatus according to the firstembodiment;

FIG. 2 is a view illustrating an example of the arrangement ofsolid-state image sensing elements on an image sensor 103;

FIG. 3A is a flowchart of processing for obtaining the DG value of eachpixel;

FIG. 3B is a flowchart of the processing for obtaining the DG value ofeach pixel;

FIG. 4A is a flowchart of processing for obtaining the DR and RB valuesof each pixel;

FIG. 4B is a flowchart of the processing for obtaining the DR and RBvalues of each pixel;

FIG. 5 is a block diagram illustrating an example of the configurationof a demosaic unit 109 according to the second embodiment;

FIG. 6A is a flowchart of processing for obtaining the DG value of eachpixel;

FIG. 6B is a flowchart of the processing for obtaining the DG value ofeach pixel;

FIG. 7 is a flowchart of processing performed by a first interpolationunit 502;

FIG. 8A is a flowchart of processing for obtaining the DG value of eachpixel;

FIG. 8B is a flowchart of the processing for obtaining the DG value ofeach pixel;

FIG. 9 is a view illustrating an example of the arrangement ofsolid-state image sensing elements on an image sensor 103;

FIG. 10A is a flowchart of processing for obtaining the DR and DB valuesof each pixel which constitutes a color image; and

FIG. 10B is a flowchart of the processing for obtaining the DR and DBvalues of each pixel which constitutes a color image.

DESCRIPTION OF EMBODIMENTS

Preferred embodiments of the present invention will be described belowwith reference to the accompanying drawings. Note that the embodimentsto be described hereinafter merely exemplify a case in which the presentinvention is actually practiced, and are practical embodiments of thearrangement defined in claims.

[First Embodiment]

An example of the functional configuration of an image sensing apparatusaccording to this embodiment will be described first with reference to ablock diagram shown in FIG. 1. Because light in the external world inwhich an object 90 is present enters an image sensor 103 via an opticalsystem 101, the image sensor 103 accumulates charges corresponding tothe light incident on it. The image sensor 103 outputs an analog imagesignal corresponding to the accumulated charges to an A/D conversionunit 104 in a subsequent stage.

The A/D conversion unit 104 converts the analog image signal input fromthe image sensor 103 into a digital image signal, and outputs theconverted digital image signal to a signal processing unit 105 and amedia I/F 107 in subsequent stages.

The signal processing unit 105 performs various types of imageprocessing (to be described later) for a color image represented by thedigital image signal input from the A/D conversion unit 104. The signalprocessing unit 105 outputs the color image having undergone the varioustypes of image processing to a display unit 106 and the media I/F 107 insubsequent stages.

The image sensor 103 will be described next. As shown in FIG. 2, theimage sensor 103 includes solid-state image sensing elements DR, DG, andDB which are two-dimensionally arranged on it. The solid-state imagesensing elements DR are used to sense R components at a firstsensitivity. The solid-state image sensing elements DG are used to senseG components at the first sensitivity. The solid-state image sensingelements DB are used to sense B components at the first sensitivity.Note that “DR” indicates dark red (red with a first brightness), “DG”indicates dark green (green with the first brightness), and “DB”indicates dark blue (blue with the first brightness). The image sensor103 also includes solid-state image sensing elements LR, LG, and LBwhich are two-dimensionally arranged on it. The solid-state imagesensing elements LR are used to sense R components at a secondsensitivity higher than the first sensitivity. The solid-state imagesensing elements LG are used to sense G components at the secondsensitivity. The solid-state image sensing elements LB are used to senseB components at the second sensitivity. Note that “LR” indicates lightred (red with a second brightness higher than the first brightness),“LG” indicates light green (green with the second brightness), and “LB”indicates light blue (blue with the second brightness).

The layout pattern of these solid-state image sensing elements arrangedon the image sensor 103 will be described in more detail herein. Asshown in FIG. 2, two-dimensional arrays each including 4×4 solid-stateimage sensing elements formed from the solid-state image sensingelements DR, DG, DB, LR, LG, and LB are repeatedly arranged on the imagesensor 103 without overlapping. In one two-dimensional array ofsolid-state image sensing elements, the ratio between the numbers ofsolid-state image sensing elements DR, DG, and DB is 1:2:1. Again inthis array, the ratio between the numbers of solid-state image sensingelements LR, LG, and LB is 1:2:1. Moreover, a column of solid-stateimage sensing elements (or a row of solid-state image sensing elements)formed from the solid-state image sensing elements DG and LG is arrangedon the image sensor 103 for every other column (row). The solid-stateimage sensing elements LR and LB can be interchanged with each other,and the solid-state image sensing elements DR and DB can similarly beinterchanged with each other.

In this manner, the image sensing apparatus according to this embodimentincludes an image sensor on which solid-state image sensing elements forsensing color components at a first sensitivity and solid-state imagesensing elements for sensing color components at a second sensitivityhigher than the first sensitivity are alternately, two-dimensionallyarranged.

The signal processing unit 105 will be described next. A camera controlunit 108 performs AE/AF/AWB control. A demosaic unit 109 generates anHDR image by performing interpolation processing for pixels sensed atthe first sensitivity and interpolation processing for pixels sensed atthe second sensitivity, in a color image represented by the digitalimage signal input from the A/D conversion unit 104.

A color processing unit 111 performs various types of color processingsuch as color balance processing, γ processing, sharpness processing,and noise reduction processing for the HDR image. The color processingunit 111 outputs the HDR image having undergone the various types ofcolor processing to the display unit 106 and media I/F 107 in subsequentstages.

Processing performed by the demosaic unit 109 to obtain the DG value ofeach pixel in the color image will be described next with reference toFIGS. 3A and 3B showing flowcharts of this processing. Note thatprocessing performed by the demosaic unit 109 to obtain the LG value ofeach pixel which constitutes the color image is processing in which “DG”is substituted by “LG” in the flowcharts shown in FIGS. 3A and 3B.

In step S301, the demosaic unit 109 secures a memory area, used toperform processing to be described later, in a memory which is providedin itself or managed by it, and initializes both the variables i and jindicating the pixel position in the above-mentioned color image tozero. Note that the variable i indicates the x-coordinate value in thecolor image, and the variable j indicates the y-coordinate value in thecolor image. Note also that the position of the upper left corner in thecolor image is defined as an origin (i,j)=(0,0). The setting of acoordinate system defined in the color image is not limited to this, asa matter of course.

In step S302, the demosaic unit 109 reads out, from the above-mentionedmemory, map information (filter array) indicating which of solid-stateimage sensing elements DR, DG, DB, LR, LG, and LB is placed at eachposition on the image sensor 103, as shown in FIG. 2.

In step S303, the demosaic unit 109 determines using the map informationwhether the solid-state image sensing element at the position on theimage sensor 103, which corresponds to the pixel position (i,j) in thecolor image, is the solid-state image sensing element DG. This amountsto determining whether the pixel at the pixel position (i,j) in thecolor image is sensed by the solid-state image sensing element DG. Thisdetermination can be done in accordance with whether the solid-stateimage sensing element at the position (i,j) on the image sensor 103 isthe solid-state image sensing element DG, upon defining the position ofthe upper left corner on the image sensor 103 as an origin.

If it is determined in step S303 that the solid-state image sensingelement corresponding to the pixel position (i,j) is the solid-stateimage sensing element DG, the process advances to step S309; otherwise,the process advances to step S304.

In step S304, the demosaic unit 109 determines using the map informationwhether the solid-state image sensing element at the position on theimage sensor 103, which corresponds to the pixel position (i−1,j−1) inthe color image, is the solid-state image sensing element DG. Thisdetermination is done in the same way as in step S303.

If it is determined in step S304 that the solid-state image sensingelement corresponding to the pixel position (i−1,j−1) is the solid-stateimage sensing element DG, the process advances to step S305; otherwise,the process advances to step S310.

In step S305, the demosaic unit 109 calculates equations presented inmathematical 1. This yields a variation evaluation value deff1 for fivepixels juxtaposed from the upper left to the lower right with the pixelat the pixel position (i,j) as the center, and a variation evaluationvalue deff2 for five pixels juxtaposed from the upper right to the lowerleft with the pixel at the pixel position (i,j) as the center. Note thatP(i,j) indicates the pixel value at the pixel position (i,j) in thecolor image.deff1=|2×P(i,j)−P(i−2,j−2)−P(i+2,j+2)|+|P(i−1,j−1)−P(i+1,j+1)|deff2=|2×P(i,j)−P(i−2,j+2)−P(i+2,j−2)|+|P(i−1,j+1)−P(i+1,j−1)|  [Mathematical 1]

In step S306, the demosaic unit 109 compares the variation evaluationvalues deff1 and deff2. If the comparison result shows deff1<deff2, theprocess advances to step S307; or if this comparison result showsdeff1≧deff2, the process advances to step S308.

In step S307, the demosaic unit 109 performs interpolation calculationusing the pixel values of pixels adjacent to the pixel at the pixelposition (i,j) to obtain DG(i,j) indicating the DG value of the pixel atthe pixel position (i,j) in accordance with an equation presented inmathematical 2.DG(i,j)=|(2×P(i,j)−P(i−2,j−2)−P(i+2,j+2))÷4|+(P(i−1,j−1)+P(i+1,j+1))÷2  [Mathematical 2]

This interpolation processing means one-dimensional lowpass filterprocessing, and the coefficient value for each pixel value in theequation presented in mathematical 2 corresponds to a filtercoefficient. On the other hand, in step S308, the demosaic unit 109performs interpolation calculation using the pixel values of pixelsadjacent to the pixel at the pixel position (i,j) to obtain DG(i,j)indicating the DG value of the pixel at the pixel position (i,j) inaccordance with an equation presented in mathematical 3.DG(i,j)=|(2×P(i,j)−P(i−2,j+2)−P(i+2,j−2))÷4|+(P(i−1,j+1)+P(i+1,j−1))÷2  [Mathematical 3]

In step S309, the demosaic unit 109 substitutes the pixel value P(i,j)for DG(i,j). In step S310, the demosaic unit 109 determines whether thevalue of the variable i is equal to “pel” (the total number of pixels inthe x direction in the color image)−1. If it is determined that i=pel−1,the process advances to step S311; or if it is determined that i≠pel−1,the value of the variable i is incremented by one and the processes instep S303 and subsequent steps are repeated.

In step S311, the demosaic unit 109 determines whether the value of thevariable j is larger than “line” (the total number of pixels in the ydirection in the color image)−1. If it is determined in step S311 thatj>line−1, the process advances to step S312; or if it is determined instep S311 that j≦line−1, the value of the variable i is initialized tozero, the value of the variable j is incremented by one, and theprocesses in step S303 and subsequent steps are repeated.

In step S312, the demosaic unit 109 initializes both the variables i andj to zero. In step S313, the demosaic unit 109 determines using the mapinformation whether the solid-state image sensing element at theposition on the image sensor 103, which corresponds to the pixelposition (i−1,j) or (i,j−1) in the color image, is the solid-state imagesensing element DG. This determination is done in the same way as instep S303.

If it is determined in step S313 that the solid-state image sensingelement corresponding to the pixel position (i−1,j) or (i,j−1) is thesolid-state image sensing element DG, the process advances to step S314;otherwise, the process advances to step S318.

In step S314, the demosaic unit 109 calculates equations presented inmathematical 4 to obtain a variation evaluation value deff3 for pixelswhich are adjacent to a pixel Q at the pixel position (i,j) vertically(in the y direction), and a variation evaluation value deff4 for pixelswhich are adjacent to the pixel Q horizontally (in the x direction).deff3=|P(i,j−1)−P(i,j+1)|deff4=|P(i−1,j)−P(i+1,j)|  [Mathematical 4]

In step S315, the demosaic unit 109 compares the variation evaluationvalues deff3 and deff4. If the comparison result shows deff3<deff4, theprocess advances to step S316; or if this comparison result showsdeff3≧deff4, the process advances to step S317.

In step S316, the demosaic unit 109 performs interpolation calculationusing the pixel values of pixels adjacent to the pixel at the pixelposition (i,j) to obtain DG(i,j) indicating the DG value of the pixel atthe pixel position (i,j) in accordance with an equation presented inmathematical 5.DG(i,j)=(P(i,j−1)+P(i,j+1))÷2   [Mathematical 5]

On the other hand, in step S317, the demosaic unit 109 performsinterpolation calculation using the pixel values of pixels adjacent tothe pixel at the pixel position (i,j) to obtain DG(i,j) indicating theDG value of the pixel at the pixel position (i,j) in accordance with anequation presented in mathematical 6.DG(i,j)=(P(i−1,j)+P(i+1,j))÷2   [Mathematical 6]

In step S318, the demosaic unit 109 determines whether the value of thevariable i is equal to pel−1. If it is determined in step S318 thati=pel−1, the process advances to step S319; or if it is determined instep S318 that i≠pel−1, the value of the variable i is incremented byone and the processes in step S313 and subsequent steps are repeated.

In step S319, the demosaic unit 109 determines whether the value of thevariable j is larger than line−1. If it is determined in step S319 thatj>line−1, the process ends and a shift to processing according toflowcharts shown in FIGS. 4A and 4B is made. On the other hand, if it isdetermined in step S319 that j≦line−1, the value of the variable i isinitialized to zero, the value of the variable j is incremented by one,and the processes in step S313 and subsequent steps are repeated.

Processing performed by the demosaic unit 109 to obtain the DR and RBvalues of each pixel which constitutes the color image will be describednext with reference to FIGS. 4A and 4B showing flowcharts of thisprocessing. Note that processing performed by the demosaic unit 109 toobtain the LR and LB values of each pixel which constitutes the colorimage is processing in which “DR” is substituted by “LR” and “DB” issubstituted by “LB” in the flowcharts shown in FIGS. 4A and 4B. Notealso that the processing according to the flowcharts shown in FIGS. 4Aand 4B follows the processing (the processing for DG and LG) accordingto the flowcharts shown in FIGS. 3A and 3B.

First, in step S401, the demosaic unit 109 secures a memory area, usedto perform processing to be described later, in a memory which isprovided in itself or managed by it, and initializes both the variablesi and j indicating the pixel position in the above-mentioned color imageto zero.

In step S402, the demosaic unit 109 reads out, from the above-mentionedmemory, map information (filter array) indicating which of solid-stateimage sensing elements DR, DG, DB, LR, LG, and LB is placed at eachposition on the image sensor 103, as shown in FIG. 2.

In step S403, the demosaic unit 109 determines using the map informationwhether the solid-state image sensing element at the position on theimage sensor 103, which corresponds to the pixel position (i,j) in thecolor image, is the solid-state image sensing element DG. Thisdetermination is done in the same way as in step S303.

If it is determined in step S403 that the solid-state image sensingelement corresponding to the pixel position (i,j) is the solid-stateimage sensing element DG, the process advances to step S404; otherwise,the process advances to step S407.

In step S404, the demosaic unit 109 determines using the map informationwhether the solid-state image sensing element at the position on theimage sensor 103, which corresponds to the pixel position (i−1,j) in thecolor image, is the solid-state image sensing element LB. Thisdetermination is done in the same way as in step S303.

If it is determined in step S404 that the solid-state image sensingelement corresponding to the pixel position (i−1,j) is the solid-stateimage sensing element LB, the process advances to step S405; otherwise,the process advances to step S406.

In step S405, the demosaic unit 109 performs interpolation calculationusing equations presented in mathematical 7. This yields DR(i,j)indicating the DR value of the pixel at the pixel position (i,j),DB(i,j) indicating the DB value of this pixel, LR(i,j) indicating the LRvalue of this pixel, and LB(i,j) indicating the LB value of this pixel.DR(i,j)=(DR(i−1,j−1)−DG(i−1,j−1)+DR(i+1,j+1)−DG(i+1,j+1))÷2DB(i,j)=(DB(i−1,j+1)−DG(i−1,j+1)+DB(i+1,j−1)−DG(i+1,j−1))÷2LR(i,j)=(LR(i+1,j)−LG(i+1,j))÷2+(LR(i−1,j+2)−LG(i−1,j+2)+LR(i−1,j−2)−LG(i−1,j−2))÷4LB(i,j)=(LB(i−1,j)−LG(i−1,j))÷2+(LB(i+1,j+2)−LG(i+1,j+2)+LB(i+1,j−2)−LG(i+1,j−2))÷4  [Mathematical 7]

In step S406, the demosaic unit 109 performs interpolation calculationusing equations presented in mathematical 8. This yields DR(i,j),DB(i,j), LR(i,j), and LB(i,j) of the pixel at the pixel position (i,j).DR(i,j)=(DR(i−1,j+1)−DG(i−1,j+1)+DR(i+1,j+1)−DG(i+1,j+1))÷2DB(i,j)=(DB(i−1,j−1)−DG(i−1,j−1)+DB(i+1,j+1)−DG(i+1,j+1))÷2LR(i,j)=(LR(i−1,j)−LG(i−1,j))÷2+(LR(i+1,j+2)−LG(i+1,j+2)+LR(i+1,j−2)−LG(i+1,j−2))÷4LB(i,j)=(LB(i+1,j)−LG(i+1,j))÷2+(LB(i−1,j+2)−LG(i−1,j+2)+LB(i−1,j−2)−LG(i−1,j−2))÷4  [Mathematical 8]

In step S407, the demosaic unit 109 determines using the map informationwhether the solid-state image sensing element at the position on theimage sensor 103, which corresponds to the pixel position (i,j) in thecolor image, is the solid-state image sensing element LR. Thisdetermination is done in the same way as in step S303.

If it is determined in step S407 that the solid-state image sensingelement corresponding to the pixel position (i,j) is the solid-stateimage sensing element LR, the process advances to step S408; otherwise,the process advances to step S409.

In step S408, the demosaic unit 109 performs interpolation calculationusing equations presented in mathematical 9. This yields DR(i,j),DB(i,j), LR(i,j), and LB(i,j) of the pixel at the pixel position (i,j).DR(i,j)=(DR(i,j+1)−DG(i,j+1))÷2+(DR(i−2,j−1)−DG(i−2,j−1)+DR(i+2,j−1)−DG(i+2,j−1))÷4DB(i,j)=(DB(i,j−1)−DG(i,j−1))÷2+(DB(i−2,j+1)−DG(i−2,j+1)+DB(i+2,j+1)−DG(i+2,j+1))÷4LR(i,j)=LR(i,j)LB(i,j)=(LB(i−2,j)−LG(i−2,j))÷4+(LB(i+2,j)−LG(i+2,j))÷4+(LB(i,j−2)−LR(i,j−2))÷4+(LB(i,j+2)−LG(i,j+2))÷4  [Mathematical 9]

In step S409, the demosaic unit 109 determines using the map informationwhether the solid-state image sensing element at the position on theimage sensor 103, which corresponds to the pixel position (i,j) in thecolor image, is the solid-state image sensing element LB. Thisdetermination is done in the same way as in step S303.

If it is determined in step S409 that the solid-state image sensingelement corresponding to the pixel position (i,j) is the solid-stateimage sensing element LB, the process advances to step S410; otherwise,the process advances to step S411.

In step S410, the demosaic unit 109 performs interpolation calculationusing equations presented in mathematical 10. This yields DR(i,j),DB(i,j), LR(i,j), and LB(i,j) of the pixel at the pixel position (i,j).DR(i,j)=(DR(i,j−1)−DG(i,j−1))÷2+(DR(i−2,j+1)−DG(i−2,j+1)+DR(i+2,j+1)−DG(i+2,j+1))÷4DB(i,j)=(DB(i,j+1)−DG(i,j+1))÷2+(DB(i−2,j−1)−DG(i−2,j−1)+DB(i+2,j−1)−DG(i+2,j−1))÷4LR(i,j)=(LR(i−2,j)−LG(i−2,j))÷4+(LR(i+2,j)−LG(i+2,j))÷4+(LR(i,j−2)−LR(i,j−2))÷4+(LR(i,j+2)−LG(i,j+2))÷4LB(i,j)=LB(i,j)   [Mathematical 10]

In step S411, the demosaic unit 109 determines using the map informationwhether the solid-state image sensing element at the position on theimage sensor 103, which corresponds to the pixel position (i,j) in thecolor image, is the solid-state image sensing element LG. Thisdetermination is done in the same way as in step S303.

If it is determined in step S411 that the solid-state image sensingelement corresponding to the pixel position (i,j) is the solid-stateimage sensing element LG, the process advances to step S412; otherwise,the process advances to step S415.

In step S412, the demosaic unit 109 determines using the map informationwhether the solid-state image sensing element at the position on theimage sensor 103, which corresponds to the pixel position (i−1,j) in thecolor image, is the solid-state image sensing element DR. Thisdetermination is done in the same way as in step S303.

If it is determined in step S412 that the solid-state image sensingelement corresponding to the pixel position (i−1,j) is the solid-stateimage sensing element DR, the process advances to step S413; otherwise,the process advances to step S414.

In step S413, the demosaic unit 109 performs interpolation calculationusing equations presented in mathematical 11. This yields DR(i,j),DB(i,j), LR(i,j), and LB(i,j) of the pixel at the pixel position (i,j).DR(i,j)=(DR(i−1,j)−DG(i−1,j))÷2+(DR(i+1,j+2)−DG(i+1,j+2)+DR(i+1,j−2)−DG(i+1,j−2))+4DB(i,j)=(DB(i+1,j)−DG(i+1,j))÷2+(DB(i−1,j+2)−DG(i−1,j+2)+DB(i−1,j−2)−DG(i−1,j−2))÷4LR(i,j)=(LR(i−1,j+1)−LG(i−1,j+1)+LR(i−1,j+1)−LG(i−1,j+1))÷2LB(i,j)=(LB(i−1,j−1)−LG(i−1,j−1)+LB(i+1,j+1)−LG(i+1,j+1))=2  [Mathematical 11]

On the other hand, in step S414, the demosaic unit 109 performsinterpolation calculation using equations presented in mathematical 12.This yields DR(i,j), DB(i,j), LR(i,j), and LB(i,j) of the pixel at thepixel position (i,j).DR(i,j)=(DR(i+1,j)−DG(i+1,j))÷2+(DR(i−1,j+2)−DG(i−1,j+2)+DR(i−1,j−2)−DG(i−1,j−2))÷4DB(i,j)=(DB(i−1,j)−DG(i−1,j))÷2+(DB(i+1,j+2)−DG(i+1,j+2)+DB(i+1,j−2)−DG(i+1,j−2))÷4LR(i,j)=(LR(i−1,j−1)−LG(i−1,j−1)+LR(i+1,j+1)−LG(i+1,j+1))÷2LB(i,j)=(LB(i−1,j+1)−LG(i−1,j+1)+LB(i−1,j+1)−LG(i−1,j+1))÷2  [Mathematical 12]

In step S415, the demosaic unit 109 determines using the map informationwhether the solid-state image sensing element at the position on theimage sensor 103, which corresponds to the pixel position (i,j) in thecolor image, is the solid-state image sensing element DR. Thisdetermination is done in the same way as in step S303.

If it is determined in step S415 that the solid-state image sensingelement corresponding to the pixel position (i,j) is the solid-stateimage sensing element DR, the process advances to step S416; otherwise,that is, if it is determined in step S415 that the solid-state imagesensing element corresponding to the pixel position (i,j) is thesolid-state image sensing element DB, the process advances to step S417.

In step S416, the demosaic unit 109 performs interpolation calculationusing equations presented in mathematical 13. This yields DR(i,j),DB(i,j), LR(i,j), and LB(i,j) of the pixel at the pixel position (i,j).DR(i,j)=DR(i,j)DB(i,j)=(DB(i−2,j)−DG(i−2,j))÷4+(DB(i+2,j)−DG(i+2,j))÷4+(DB(i,j−2)−DR(i,j−2))÷4+(DB(i,j+2)−DG(i,j+2))÷4LR(i,j)=(LR(i,j−1)−LG(i,j−1))÷2+(LR(i−2,j+1)−LG(i−2,j+1)+LR(i+2,j+1)−LG(i+2,j+1))÷4LB(i,j)=(LB(i,j+1)−LG(i,j+1))÷2+(LB(i−2,j−1)−LG(i−2,j−1)+LB(i+2,j−1)−LG(i+2,j−1))÷4  [Mathematical 13]

On the other hand, in step S417, the demosaic unit 109 performsinterpolation calculation using equations presented in mathematical 14.This yields DR(i,j), DB(i,j), LR(i,j), and LB(i,j) of the pixel at thepixel position (i,j).DR(i,j)=(DR(i−2,j)−DG(i−2,j))÷4+(DR(i+2,j)−DG(i+2,j))÷4+(DR(i,j−2)−DG(i,j−2))÷4+(DR(i,j+2)−DG(i,j+2))÷4DB(i,j)=DB(i,j)LR(i,j)=(LR(i,j+1)−LG(i,j+1))÷2+(LR(i−2,j−1)−LG(i−2,j−1)+LR(i+2,j−1)−LG(i+2,j−1))÷4LB(i,j)=(LB(i,j−1)−LG(i,j−1))÷2+(LB(i−2,j+1)−LG(i−2,j+1)+LB(i+2,j+1)−LG(i+2,j+1))÷4  [Mathematical 14]

In step S418, the demosaic unit 109 determines whether the value of thevariable i is equal to pel−1. If it is determined in step S418 thati=pel−1, the process advances to step S419; or if it is determined instep S418 that i≠pel−1, the value of the variable i is incremented byone and the processes in step S403 and subsequent steps are repeated.

In step S419, the demosaic unit 109 determines whether the value of thevariable j is larger than line−1. If it is determined in step S419 thatj>line−1, the process ends. On the other hand, if it is determined instep S419 that j≦line−1, the value of the variable i is initialized tozero, the value of the variable j is incremented by one, and theprocesses in step S403 and subsequent steps are repeated.

In this manner, the color components (DR, DG, DB, LR, LG, and LB) foreach pixel which constitutes the color image are determined byperforming processing according to the flowcharts shown in FIGS. 3A, 3B,4A, and 4B described above. The feature of this embodiment lies in thatthis determination processing is realized by the following calculationprocessing. That is, a color component of a given pixel sensed at thefirst sensitivity is obtained by interpolation calculation (firstcalculation) using a color component of a pixel which is adjacent to thegiven pixel and is sensed at the first sensitivity. A color component ofa given pixel sensed at the second sensitivity is obtained byinterpolation calculation (second calculation) using a color componentof a pixel which is adjacent to the given pixel and is sensed at thesecond sensitivity.

In this manner, according to this embodiment, even when pixels withdifferent sensitivities are arranged on the same sensor, the resolutionof a portion incapable of being sampled can be partially restored byperforming interpolation using the correlation between a color filterwith a highest resolution and multiple colors. Also, because demosaicingis performed based only on a pixel with one sensitivity, a stableresolution can be obtained regardless of saturation or unsaturation.

[Second Embodiment]

This embodiment is different from the first embodiment only in theconfiguration and operation of the demosaic unit 109. A demosaic unit109 according to this embodiment has a configuration as shown in FIG. 5.A saturation determination unit 501 determines, for each pixel whichconstitutes a color image, whether the pixel value is saturated. A firstinterpolation unit 502 processes a pixel with a saturated pixel value,and a second interpolation unit 503 processes a pixel with anunsaturated pixel value.

Since the operation of the second interpolation unit 503 is the same asthat of the demosaic unit 109, having been described in the firstembodiment, only the operation of the first interpolation unit 502 willbe mentioned below, and that of the second interpolation unit 503 willnot be described. Also, only differences from the first embodiment willbe mentioned below, and the second embodiment is the same as the firstembodiment except for points to be described hereinafter.

Processing with which the demosaic unit 109 according to this embodimentobtains the DG value of each pixel which constitutes a color image willbe described with reference to FIGS. 6A and 6B showing flowcharts ofthis processing. Note that the following description assumes that apixel value P(i,j) is stored in advance for DG(i,j).

In step S601, the demosaic unit 109 secures a memory area, used toperform processing to be described later, in a memory which is providedin itself or managed by it, and initializes both the variables i and jindicating the pixel position in the above-mentioned color image tozero.

In step S602, the demosaic unit 109 reads out, from the above-mentionedmemory, map information (filter array) indicating which of solid-stateimage sensing elements DR, DG, DB, LR, LG, and LB is placed at eachposition on an image sensor 103, as shown in FIG. 2.

In step S603, the demosaic unit 109 determines using the map informationwhether the solid-state image sensing element at the position on theimage sensor 103, which corresponds to the pixel position (i,j) in thecolor image, is the solid-state image sensing element LG. Thisdetermination is done in the same way as in step S303.

If it is determined in step S603 that the solid-state image sensingelement corresponding to the pixel position (i,j) is the solid-stateimage sensing element LG, the process advances to step S604; otherwise,the process advances to step S607.

In step S604, the demosaic unit 109 determines whether the pixel valueof the pixel at the pixel position (i,j) in the color image issaturated. If it is determined in step S604 that this pixel value issaturated, the process advances to step S605; otherwise, the processadvances to step S606. Determination as to whether the pixel value issaturated is done in the following way. That is, if the pixel value isequal to or larger than a predetermined value, it is determined thatthis pixel value is saturated; or if the pixel value is smaller than thepredetermined value, it is determined that this pixel value isunsaturated. Although this “predetermined value” is not limited to aspecific value, the following description assumes that the maximum valueof a sensor analog value is used for the sake of convenience.

In step S605, the demosaic unit 109 operates the first interpolationunit 502, so the first interpolation unit 502 performs interpolationprocessing (to be described later) to obtain the DG value=DG(i,j) of thepixel at the pixel position (i,j). Processing performed by the firstinterpolation unit 502 in this step will be described in detail later.

In step S606, the demosaic unit 109 operates the second interpolationunit 503, so the second interpolation unit 503 performs the sameoperation as that of the demosaic unit 109, having been described in thefirst embodiment, to obtain the DG value=DG(i,j) of the pixel at thepixel position (i,j).

In step S607, the demosaic unit 109 determines whether the value of thevariable i is equal to pel−1. If it is determined in step S607 thati=pel−1, the process advances to step S608; or if it is determined instep S607 that i≠pel−1, the value of the variable i is incremented byone and the processes in step S603 and subsequent steps are repeated.

In step S608, the demosaic unit 109 determines whether the value of thevariable j is larger than line−1. If it is determined in step S608 thatj>line−1, the process advances to step S609. On the other hand, if it isdetermined in step S608 that j≦line−1, the value of the variable i isinitialized to zero, the value of the variable j is incremented by one,and the processes in step S603 and subsequent steps are repeated.

In step S609, the demosaic unit 109 initializes both the variables i andj to zero. In step S610, the demosaic unit 109 determines using the mapinformation whether the solid-state image sensing element at theposition on the image sensor 103, which corresponds to the pixelposition (i,j) in the color image, is the solid-state image sensingelement DR or DB. This determination is done in the same way as in stepS303 mentioned above.

If it is determined in step S610 that the solid-state image sensingelement corresponding to the pixel position (i,j) is the solid-stateimage sensing element DR or DB, the process advances to step S611;otherwise, the process advances to step S614.

In step S611, the demosaic unit 109 determines whether the pixel valueof the pixel at the pixel position (i−1,j) or (i+1,j) in the color imageis saturated. If it is determined in step S611 that this pixel value issaturated, the process advances to step S612; otherwise, the processadvances to step S613.

In step S612, the demosaic unit 109 operates the first interpolationunit 502, so the first interpolation unit 502 performs interpolationprocessing (to be described later) to obtain the DG value=DG(i,j) of thepixel at the pixel position (i,j). Processing performed by the firstinterpolation unit 502 in this step will be described in detail later.

In step S613, the demosaic unit 109 operates the second interpolationunit 503, so the second interpolation unit 503 performs the sameoperation as that of the demosaic unit 109, having been described in thefirst embodiment, to obtain the DG value=DG(i,j) of the pixel at thepixel position (i,j).

In step S614, the demosaic unit 109 determines using the map informationwhether the solid-state image sensing element at the position on theimage sensor 103, which corresponds to the pixel position (i,j) in thecolor image, is the solid-state image sensing element LR or LB. Thisdetermination is done in the same way as in step S303 mentioned above.

If it is determined in step S614 that the solid-state image sensingelement corresponding to the pixel position (i,j) is the solid-stateimage sensing element LR or LB, the process advances to step S615;otherwise, the process advances to step S618.

In step S615, the demosaic unit 109 determines whether the pixel valueof the pixel at the pixel position (i−1,j−1), (i−1,j+1), (i+1,j−1), or(i+1,j+1) in the color image is saturated. If it is determined in stepS615 that this pixel value is saturated, the process advances to stepS616; otherwise, the process advances to step S617.

In step S616, the demosaic unit 109 operates the first interpolationunit 502, so the first interpolation unit 502 performs interpolationprocessing (to be described later) to obtain the DG value=DG(i,j) of thepixel at the pixel position (i,j). Processing performed by the firstinterpolation unit 502 in this step will be described in detail later.

In step S617, the demosaic unit 109 operates the second interpolationunit 503, so the second interpolation unit 503 performs the sameoperation as that of the demosaic unit 109, having been described in thefirst embodiment, to obtain the DG value=DG(i,j) of the pixel at thepixel position (i,j).

In step S618, the demosaic unit 109 determines whether the value of thevariable i is equal to pel−1. If it is determined in step S618 thati=pel−1, the process advances to step S619; or if it is determined instep S618 that i≠pel−1, the value of the variable i is incremented byone and the processes in step S610 and subsequent steps are repeated.

In step S619, the demosaic unit 109 determines whether the value of thevariable j is larger than line−1. If it is determined in step S619 thatj>line−1, the process ends. On the other hand, if it is determined instep S619 that j≦line−1, the value of the variable i is initialized tozero, the value of the variable j is incremented by one, and theprocesses in step S610 and subsequent steps are repeated.

Processing performed by the first interpolation unit 502 will bedescribed with reference to FIG. 7 showing a flowchart of thisprocessing. In step S703, the first interpolation unit 502 determinesusing the map information whether the solid-state image sensing elementat the position on the image sensor 103, which corresponds to the pixelposition (i,j) in the color image, is the solid-state image sensingelement LG. This determination is done in the same way as in step S303mentioned above.

If it is determined in step S703 that the solid-state image sensingelement corresponding to the pixel position (i,j) is the solid-stateimage sensing element LG, the process advances to step S704; otherwise,the process advances to step S705. In step S704, the first interpolationunit 502 calculates an equation presented in mathematical 15 todetermine DG(i,j).DG(i,j)=α×LG(i,j)   [Mathematical 15]

where α is a constant (0<α≦1) which represents the gain and is set inadvance.

In step S705, the first interpolation unit 502 determines using the mapinformation whether the solid-state image sensing element at theposition on the image sensor 103, which corresponds to the pixelposition (i,j) in the color image, is the solid-state image sensingelement DR or DB. This determination is done in the same way as in stepS303 mentioned above.

If it is determined in step S705 that the solid-state image sensingelement corresponding to the pixel position (i,j) is the solid-stateimage sensing element DR or DB, the process advances to step S706;otherwise, the process ends. In step S706, the first interpolation unit502 calculates equations presented in mathematical 16. This calculationyields an evaluation value deff5−1, variation evaluation value deff6−1,and evaluation value deff7−1. The evaluation value deff5−1 is anevaluation value for pixels adjacent to the pixel position (i,j) on theupper left and lower right sides. The variation evaluation value deff6−1is a variation evaluation value for pixels adjacent to the pixelposition (i,j) on the right and left sides. The evaluation value deff7−1is an evaluation value for pixels adjacent to the pixel position (i,j)on the lower left and upper right sides. Note that if the calculationresult of each evaluation value shows that the pixel value of one of thepair of upper left and lower right pixels is saturated, an evaluationvalue deff5−2 is obtained in place of the evaluation value deff5−1.Also, if the pixel value of one of the pair of right and left pixels issaturated, an evaluation value deff6−2 is obtained in place of theevaluation value deff6−1. Moreover, if the pixel value of one of thepair of lower left and upper right pixels is saturated, an evaluationvalue deff7−2 is obtained in place of the evaluation value deff7−1. Notethat MAX is the difference between the minimum and maximum pixel values,and is 255 for a pixel value with 8 bits and 65535 for a pixel valuewith 16 bits.deff5−1=|P(i−1,j−1)−P(i+1,j+1)|deff5−2=MAXdeff6−1=|P(i−1,j)−P(i+1,j)|deff6−2=MAXdeff7−1=|P(i−1,j+1)−P(i+1,j−1)|deff7−2=MAX   [Mathematical 16]

In the following description, the obtained one of deff5−1 and deff5−2will be represented as deff5. Similarly, the obtained one of deff6−1 anddeff6−2 will be represented as deff6. Again similarly, the obtained oneof deff7−1 and deff7−2 will be represented as deff7. In step S706, thefirst interpolation unit 502 compares the evaluation values deff5,deff6, and deff7. If the comparison result shows that both conditions:deff5<deff6 and deff5<deff7 are satisfied, the process advances to stepS707; otherwise, the process advances to step S708.

In step S707, the first interpolation unit 502 performs interpolationcalculation using the pixel values of pixels adjacent to the pixelposition (i,j) on the upper left and lower right sides to obtain DG(i,j)of the pixel at the pixel position (i,j) in accordance with an equationpresented in mathematical 17.DG(i,j)=(P(i−1,j−1)+P(i+1,j+1))+2   [Mathematical 17]

In step S708, the first interpolation unit 502 compares the evaluationvalues deff6 and deff7. If the comparison result shows deff6<deff7, theprocess advances to step S709; or if this comparison result showsdeff6≧deff7, the process advances to step S710.

In step S709, the first interpolation unit 502 performs interpolationcalculation using the pixel values of pixels adjacent to the pixelposition (i,j) on the right and left sides to obtain DG(i,j) of thepixel at the pixel position (i,j) in accordance with an equationpresented in mathematical 18.DG(i,j)=(P(i−1,j)+P(i+1,j))÷2   [Mathematical 18]

In step S710, the first interpolation unit 502 performs interpolationcalculation using the pixel values of pixels adjacent to the pixelposition (i,j) on the lower left and upper right sides to obtain DG(i,j)of the pixel at the pixel position (i,j) in accordance with an equationpresented in mathematical 19.DG(i,j)=(P(i−1,j+1)+P(i+1,j−1))÷2   [Mathematical 19]

In step S711, the first interpolation unit 502 performs interpolationcalculation using the pixel values of pixels adjacent to the pixelposition (i,j) on the right and left sides to obtain DG(i,j) of thepixel at the pixel position (i,j) in accordance with an equationpresented in mathematical 20.DG(i,j)=(P(i−1,j)+P(i+1,j))÷2   [Mathematical 20]

In this manner, according to this embodiment, even when pixels withdifferent sensitivities are arranged on the same sensor, the resolutionof a portion incapable of being sampled can be partially restored byperforming interpolation using the correlation between a color filterwith a highest resolution and multiple colors.

Also, because pixel value determination which uses two pixels withdifferent sensitivities is performed for an unsaturated pixel value, itis possible to obtain a higher resolution. Moreover, even if eitherpixel value is saturated, it is possible to perform pixel interpolation,which improves the resolution.

[Third Embodiment]

Another embodiment of the demosaic unit 109 according to the secondembodiment will be described in the third embodiment. Note that in thisembodiment, solid-state image sensing elements arranged on an imagesensor 103 preferably have a layout distribution shown in FIG. 9. Also,only differences from the second embodiment will be mentioned below, andthe third embodiment is the same as the first embodiment except forpoints to be described hereinafter.

Processing with which a demosaic unit 109 according to this embodimentobtains the DG value of each pixel which constitutes a color image willbe described with reference to FIGS. 8A and 8B showing flowcharts ofthis processing. Note that the following description assumes that apixel value P(i,j) is stored in advance for DG(i,j).

In step S801, the demosaic unit 109 secures a memory area, used toperform processing to be described later, in a memory which is providedin itself or managed by it, and initializes both the variables i and jindicating the pixel position in the above-mentioned color image tozero.

In step S802, the demosaic unit 109 reads out, from the above-mentionedmemory, map information (filter array) indicating which of solid-stateimage sensing elements DR, DG, DB, LR, LG, and LB is placed at eachposition on the image sensor 103, as shown in FIG. 9.

In step S803, the demosaic unit 109 determines using the map informationwhether the solid-state image sensing element at the position on theimage sensor 103, which corresponds to the pixel position (i,j) in thecolor image, is the solid-state image sensing element DG. Thisdetermination is done in the same way as in step S303 mentioned above.

If it is determined in step S803 that the solid-state image sensingelement corresponding to the pixel position (i,j) is not the solid-stateimage sensing element DG, the process advances to step S804; otherwise,the process advances to step S811.

In step S804, the demosaic unit 109 determines using the map informationwhether the solid-state image sensing element at the position on theimage sensor 103, which corresponds to the pixel position (i,j) in thecolor image, is the solid-state image sensing element LG. Thisdetermination is done in the same way as in step S303 mentioned above.

If it is determined in step S804 that the solid-state image sensingelement corresponding to the pixel position (i,j) is not the solid-stateimage sensing element LG, the process advances to step S812; otherwise,the process advances to step S805.

In step S805, the demosaic unit 109 determines whether the pixel valueof the pixel at the pixel position (i,j) in the color image issaturated. If it is determined in step S805 that this pixel value issaturated, the process advances to step S806; otherwise, the processadvances to step S810.

In step S806, a first interpolation unit 502 calculates equationspresented in mathematical 21. This calculation yields an evaluationvalue deff8 for pixels adjacent to the pixel position (i,j) on the upperleft and lower right sides, and a variation evaluation value deff9 forpixels adjacent to the pixel position (i,j) on the lower left and upperright sides.deff8=|P(i−1,j−1)−P(i+1,j+1)|deff9=|P(i−1,j+1)−P(i+1,j−1)|  [Mathematical 21]

In step S807, the evaluation values deff8 and deff9 are compared witheach other. If deff8<deff9, the process advances to step S808; or ifdeff8≧deff9, the process advances to step S809.

In step S808, the first interpolation unit 502 performs interpolationcalculation using an equation presented in mathematical 22. This yieldsDG(i,j) of the pixel at the pixel position (i,j).DG(i,j)=(P(i−1,j−1)+P(i+1,j+1))÷2   [Mathematical 22]

In step S809, the first interpolation unit 502 performs interpolationcalculation using an equation presented in mathematical 23. This yieldsDG(i,j) of the pixel at the pixel position (i,j).DG(i,j)=(P(i−1,j+1)+P(i+1,j−1))÷2   [Mathematical 23]

In step S810, the first interpolation unit 502 performs interpolationcalculation using an equation presented in mathematical 24. This yieldsDG(i,j) of the pixel at the pixel position (i,j).DG(i,j)=α×P(i,j)   [Mathematical 24]

In step S811, the demosaic unit 109 substitutes the pixel value P(i,j)for DG(i,j). In step S812, the demosaic unit 109 determines whether thevalue of the variable i is equal to pel−1. If it is determined in stepS812 that i=pel−1, the process advances to step S813; or if it isdetermined in step S812 that i≠pel−1, the value of the variable i isincremented by one and the processes in step S803 and subsequent stepsare repeated.

In step S813, the demosaic unit 109 determines whether the value of thevariable j is larger than line−1. If it is determined in step S813 thatj>line−1, the process advances to step S814. On the other hand, if it isdetermined in step S813 that j≦line−1, the value of the variable i isinitialized to zero, the value of the variable j is incremented by one,and the processes in step S803 and subsequent steps are repeated.

In step S814, the demosaic unit 109 initializes both the variables i andj to zero. In step S815, the demosaic unit 109 determines whether thepixel position (i,j) corresponds to DG or LG. If it is determined thatthe pixel position (i,j) corresponds to DG or LG, the process advancesto step S825; otherwise, the process advances to step S816.

In step S816, the demosaic unit 109 determines whether the solid-stateimage sensing element at the position on the image sensor 103, whichcorresponds to the pixel position (i−2,j), (i+2,j), (i,j), (i,j−2), or(i,j+2), is saturated. This determination is done in the same way as instep S805. If it is determined in step S816 that this solid-state imagesensing element is saturated, the process advances to step S817;otherwise, the process advances to step S821.

In step S817, the first interpolation unit 502 calculates equationspresented in mathematical 25. This calculation yields an evaluationvalue deff10 for pixels adjacent to the pixel position (i,j) on theupper and lower sides, and a variation evaluation value deff11 forpixels adjacent to the pixel position (i,j) on the right and left sides.deff10=|P(i,j−1)−P(i,j+1 )|deff11=|P(i−1,j)−P(i+1,j)|  [Mathematical 25]

In step S818, the demosaic unit 109 compares the evaluation valuesdeff10 and deff11. If the comparison result shows deff10<deff11, theprocess advances to step S819; or if this comparison result showsdeff10≧deff11, the process advances to step S820.

In step S819, the first interpolation unit 502 performs interpolationcalculation using an equation presented in mathematical 26. This yieldsDG(i,j) of the pixel at the pixel position (i,j).DG(i,j)=(P(i,j−1)+P(i,j+1))÷2   [Mathematical 26]

In step S820, the first interpolation unit 502 performs interpolationcalculation using an equation presented in mathematical 27. This yieldsDG(i,j) of the pixel at the pixel position (i,j).DG(i,j)=(P(i−1,j)+P(i+1,j))÷2   [Mathematical 27]

In step S821, a second interpolation unit 503 calculates equationspresented in mathematical 28. This calculation yields an evaluationvalue deff12 for pixels which are adjacent to the pixel position (i,j)vertically, and a variation evaluation value deff13 for pixels which areadjacent to the pixel position (i,j) horizontally.deff12=|2×(P(i,j)−P(i,j−2)−P(i,j+2))|+|P(i,j−1)−P(i,j+1)|deff13=|2×(P(i,j)−P(i−2,j)−P(i+2,j))|+|P(i−1,j)−P(i+1,j)|  [Mathematical28]

In step S822, the demosaic unit 109 compares the evaluation valuesdeff12 and deff13. If the comparison result shows deff12<deff13, theprocess advances to step S823; or if this comparison result showsdeff12≧deff13, the process advances to step S824.

In step S823, the second interpolation unit 503 performs interpolationcalculation using an equation presented in mathematical 29. This yieldsDG(i,j) of the pixel at the pixel position (i,j).DG(i,j)=|2×(P(i,j)−P(i,j−2)−P(i,j+2))|÷4+(P(i,j−1)+P(i,j+1))÷2  [Mathematical 29]

In step S824, the second interpolation unit 503 performs interpolationcalculation using an equation presented in mathematical 30. This yieldsDG(i,j) of the pixel at the pixel position (i,j).DG(i,j)=|2×(P(i,j)−P(i−2,j)−P(i+2,j))|÷4+(P(i−1,j)+P(i+1,j))=2  [Mathematical 30]

In step S825, the demosaic unit 109 determines whether the value of thevariable i is equal to pel−1. If it is determined in step S825 thati=pel−1, the process advances to step S826; or if it is determined instep S825 that i≠pel−1, the value of the variable i is incremented byone and the processes in step S815 and subsequent steps are repeated.

In step S826, the demosaic unit 109 determines whether the value of thevariable j is larger than line−1. If it is determined in step S826 thatj>line−1, the process ends. On the other hand, if it is determined instep S826 that j≦line−1, the value of the variable i is initialized tozero, the value of the variable j is incremented by one, and theprocesses in step S815 and subsequent steps are repeated.

Processing with which the demosaic unit 109 according to this embodimentobtains the DR and DB values of each pixel which constitutes a colorimage will be described with reference to FIGS. 10A and 10B showingflowcharts of this processing.

In step S1001, the demosaic unit 109 secures a memory area, used toperform processing to be described later, in a memory which is providedin itself or managed by it, and initializes both the variables i and jindicating the pixel position in the above-mentioned color image tozero.

In step S1002, the demosaic unit 109 reads out, from the above-mentionedmemory, map information (filter array) indicating which of solid-stateimage sensing elements DR, DG, DB, LR, LG, and LB is placed at eachposition on the image sensor 103, as shown in FIG. 9.

In step S1003, the demosaic unit 109 determines using the mapinformation whether the solid-state image sensing element at theposition on the image sensor 103, which corresponds to the pixelposition (i,j) in the color image, is the solid-state image sensingelement DG. This determination is done in the same way as in step S303mentioned above. If it is determined that the pixel position (i,j)corresponds to DG, the process advances to step S1004; otherwise, theprocess advances to step S1007.

In step S1004, the demosaic unit 109 determines whether the pixel valueof the pixel at the pixel position (i−1,j), (i+1,j), (i,j−1), or (i,j+1)in the color image is saturated. If it is determined in step S1004 thatthis pixel value is saturated, the process advances to step S1005;otherwise, the process advances to step S1006.

In step S1005, the first interpolation unit 502 performs interpolationcalculation using equations presented in mathematical 31. This yieldsDR(i,j) and DB(i,j) of the pixel at the pixel position (i,j).when P(i−1,j) corresponds to LBDR(i,j)=(DR(i,j−1)−DR(i,j−1))÷2+(DR(i−2,j+1)−DG(i−2,j+1)+DR(i+2,j+1)−DG(i+2,j+1))÷4DB(i,j)=(DB(i+1,j)−DG(i+1,j))÷2+(DB(i−1,j−2)−DG(i−1,j−2)+DB(i−1,j+2)−DG(i−1,j+2))÷4when P(i−1,j) corresponds to DBDR(i,j)=(DR(i,j+1)−DG(i,j+1))÷2+(DR(i−2,j−1)−DG(i−2,j−1)+DR(i+2,j−1)−DG(i+2,j−1))÷4DB(i,j)=(DB(i−1,j)−DG(i−1,j))÷2+(DB(i+1,j−2)−DG(i+1,j−2)+DB(i+1,j+2)−DG(i+1,j+2))÷4  [Mathematical 31]

In step S1006, the second interpolation unit 503 performs interpolationcalculation using equations presented in mathematical 32. This yieldsDR(i,j) and DB(i,j) of the pixel at the pixel position (i,j).when P(i−1,j) corresponds to LBDR(i,j)=(LR(i,j−1)−DR(i,j+1))÷2DB(i,j)=(LB(i−1,j)−DB(i+1,j))÷2when P(i−1,j) corresponds to DBDR(i,j)=(DR(i,j−1)−LR(i,j+1))÷2DB(i,j)=(DB(i−1,j)−LB(i+1,j))+2   [Mathematical 32]

In step S1007, the demosaic unit 109 determines using the mapinformation whether the solid-state image sensing element at theposition on the image sensor 103, which corresponds to the pixelposition (i,j) in the color image, is the solid-state image sensingelement LR. This determination is done in the same way as in step S303mentioned above. If it is determined that the pixel position (i,j)corresponds to LR, the process advances to step S1008; otherwise, theprocess advances to step S1011.

In step S1008, the demosaic unit 109 determines whether the pixel valueof the pixel at the pixel position (i,j) in the color image issaturated. If it is determined in step S1008 that this pixel value issaturated, the process advances to step S1009; otherwise, the processadvances to step S1010.

In step S1009, the first interpolation unit 502 performs interpolationcalculation using equations presented in mathematical 33. This yieldsDR(i,j) and DB(i,j) of the pixel at the pixel position (i,j).DR(i,j)=(DR(i−2,j)−DG(i−2,j))÷4+(DR(i+2,j)−DG(i+2,j))÷4+(DR(i,j−2)−DR(i,j−2))÷4+(DR(i,j+2)−DG(i,j+2))÷4DB(i,j)=(DB(i−1,j−1)+DB(i+1,j+1))÷2   [Mathematical 33]

In step S1010, the second interpolation unit 503 performs interpolationcalculation using equations presented in mathematical 34. This yieldsDR(i,j) and DB(i,j) of the pixel at the pixel position (i,j).DR(i,j)=P(i,j)×αDB(i,j)=(DB(i−1,j−1)+DB(i+1,j+1))÷2   [Mathematical 34]

In step S1011, the demosaic unit 109 determines using the mapinformation whether the solid-state image sensing element at theposition on the image sensor 103, which corresponds to the pixelposition (i,j) in the color image, is the solid-state image sensingelement LB. This determination is done in the same way as in step S303mentioned above. If it is determined that the pixel position (i,j)corresponds to LB, the process advances to step S1012; otherwise, theprocess advances to step S1015.

In step S1012, the demosaic unit 109 determines whether the pixel valueof the pixel at the pixel position (i,j) in the color image issaturated. If it is determined in step S1012 that this pixel value issaturated, the process advances to step S1013; otherwise, the processadvances to step S1014.

In step S1013, the first interpolation unit 502 performs interpolationcalculation using equations presented in mathematical 35. This yieldsDR(i,j) and DB(i,j) of the pixel at the pixel position (i,j).DR(i,j)=(DR(i−1,j−1)+DR(i+1,j+1))÷2DB(i,j)=(DB(i−2,j)−DG(i−2,j))÷4+(DB(i+2,j)−DG(i+2,j))÷4+(DB(i,j−2)−DG(i,j−2))÷4+(DB(i,j+2)−DG(i,j+2))÷4  [Mathematical 35]

In step S1014, the second interpolation unit 503 performs interpolationcalculation using equations presented in mathematical 36. This yieldsDR(i,j) and DB(i,j) of the pixel at the pixel position (i,j).DR(i,j)=(DR(i−1,j−1)+DR(i+1,j+1))÷2DB(i,j)=P(i,j)×α  [Mathematical 36]

In step S1015, the demosaic unit 109 determines using the mapinformation whether the solid-state image sensing element at theposition on the image sensor 103, which corresponds to the pixelposition (i,j) in the color image, is the solid-state image sensingelement LG. This determination is done in the same way as in step S303mentioned above. If it is determined that the pixel position (i,j)corresponds to LG, the process advances to step S1016; otherwise, theprocess advances to step S1019.

In step S1016, the demosaic unit 109 determines whether the pixel valueof the pixel at the pixel position (i−1,j), (i+1,j), (i,j−1), or (i,j+1)in the color image is saturated. If it is determined in step S1016 thatthis pixel value is saturated, the process advances to step S1017;otherwise, the process advances to step S1018.

In step S1017, the first interpolation unit 502 performs interpolationcalculation using equations presented in mathematical 37. This yieldsDR(i,j) and DB(i,j) of the pixel at the pixel position (i,j).when P(i−1,j) corresponds to LRDR(i,j)=(DR(i+1,j)−DG(i+1,j))÷2+(DR(i−1,j−2)−DG(i−1,j−2)+DR(i−1,j+2)−DG(i−1,j+2))÷4DB(i,j)=(DB(i,j−1)−DR(i,j−1))÷2+(DB(i−2,j+1)−DG(i−2,j+1)+DB(i+2,j+1)−DG(i+2,j+1))÷4when P(i−1,j) corresponds to DRDR(i,j)=(DR(i−1,j)−DG(i−1,j))÷2+(DR(i+1,j−2)−DG(i+1,j−2)+DR(i+1,j+2)−DG(i+1,j+2))÷4DB(i,j)=(DB(i,j+1)−DG(i,j+1))÷2+(DB(i−2,j−1)−DG(i−2,j−1)+DB(i+2,j−1)−DG(i+2,j−1))÷4  [Mathematical 37]

In step S1018, the second interpolation unit 503 performs interpolationcalculation using equations presented in mathematical 38. This yieldsDR(i,j) and DB(i,j) of the pixel at the pixel position (i,j).when P(i−1,j) corresponds to LRDR(i,j)=(LR(i,j−1)−DR(i,j+1))÷2DB(i,j)=(DB(i−1,j)−LB(i+1,j))÷2when P(i−1,j) corresponds to DRDR(i,j)=(DR(i,j−1)−LR(i,j+1))÷2DB(i,j)=(LB(i−1,j)−DB(i+1,j))÷2   [Mathematical 38]

In step S1019, the demosaic unit 109 determines using the mapinformation whether the solid-state image sensing element at theposition on the image sensor 103, which corresponds to the pixelposition (i,j) in the color image, is the solid-state image sensingelement DR. This determination is done in the same way as in step S303mentioned above. If it is determined that the pixel position (i,j)corresponds to DR, the process advances to step S1020; otherwise, theprocess advances to step S1021.

In step S1020, the first interpolation unit 502 performs interpolationcalculation using equations presented in mathematical 39. This yieldsDR(i,j) and DB(i,j) of the pixel at the pixel position (i,j).DR(i,j)=P(i,j)DB(i,j)=(DB(i−1,j+1)+DB(i−1,j+1))÷2   [Mathematical 39]

In step S1021, the first interpolation unit 502 performs interpolationcalculation using equations presented in mathematical 40. This yieldsDR(i,j) and DB(i,j) of the pixel at the pixel position (i,j).DR(i,j)=(DR(i−1,j+1)+DR(i−1,j+1))÷2DB(i,j)=P(i,j)

As has been described above, according to this embodiment, even whenpixels with different sensitivities are arranged on the same sensor, theresolution of a portion incapable of being sampled can be partiallyrestored by performing interpolation using the correlation between acolor filter with a highest resolution and multiple colors.

Also, because pixel value determination which uses two pixels withdifferent sensitivities is performed for an unsaturated pixel value, itis possible to obtain a higher resolution. Moreover, even if eitherpixel value is saturated, it is possible to perform pixel interpolation,which improves the resolution.

[Other Embodiments]

The present invention can also be practiced by executing the followingprocessing. That is, software (program) which implements the functionsof the above-described embodiments is supplied to a system or apparatusvia a network or various kinds of storage media, and read out andexecuted by a computer (or, for example, a CPU or an MPU) of the systemor apparatus.

The present invention is not limited to the above-described embodiments,and various changes and modifications can be made within the spirit andscope of the present invention. Therefore, to apprise the public of thescope of the present invention, the following claims are made.

The invention claimed is:
 1. An image sensing apparatus including animage sensor on which solid-state image sensing elements which sensecolor components at a first sensitivity and solid-state image sensingelements which sense color components at a second sensitivity higherthan the first sensitivity are alternately, two-dimensional arranged,comprising: a first calculation unit that obtains a color component of agiven pixel, sensed at the first sensitivity, by performinginterpolation calculation using color components of pixels each of whichis adjacent to the given pixel and is sensed at the first sensitivity,in a color image based on an image signal output from the image sensor;a second calculation unit that obtains a color component of a givenpixel, sensed at the second sensitivity, by performing interpolationcalculation using color components of pixels each of which is adjacentto the given pixel and is sensed at the second sensitivity, in the colorimage output from the image sensor; and a unit that outputs the colorimage in which color components of all pixels are determined by saidfirst calculation unit and said second calculation unit, wherein thesolid-state image sensing elements which sense the color components atthe first sensitivity include solid-state image sensing elements DRwhich sense R components at a first brightness, solid-state imagesensing elements DG which sense G components at the first brightness,solid-state image sensing elements DB which sense B components at thefirst brightness, wherein the solid-state image sensing elements whichsense the color components at the second sensitivity include solid-stateimage sensing elements LR which sense R components at a secondbrightness higher than the first brightness, solid-state image sensingelements LG which sense G components at the second brightness, andsolid-state image sensing elements LB which sense B components at thesecond brightness, and wherein said first calculation unit obtains an Rcomponent, having the first brightness, of a pixel, sensed by thesolid-state image sensing element DG, by performing interpolationcalculation using an R component and a G component, both having thefirst brightness, of a pixel adjacent to the pixel sensed by thesolid-state image sensing element DG, and obtains a B component, havingthe first brightness, of the pixel, sensed by the solid-state imagesensing element DG, by performing interpolation calculation using a Bcomponent and a G component, both having the first brightness, of apixel adjacent to the pixel sensed by the solid-state image sensingelement DG, and said second calculation unit obtains an R component,having the second brightness, of the pixel, sensed by the solid-stateimage sensing element DG, by performing interpolation calculation usingan R component and a G component, both having the second brightness, ofa pixel adjacent to the pixel sensed by the solid-state image sensingelement DG, and obtains a B component, having the second brightness, ofthe pixel, sensed by the solid-state image sensing element DG, byperforming interpolation calculation using a B component and a Gcomponent, both having the second brightness, of a pixel adjacent to thepixel sensed by the solid-state image sensing element DG.
 2. The imagesensing apparatus according to claim 1, wherein on the image sensor, acolumn of solid-state image sensing elements formed from the solid-stateimage sensing elements DG and the solid-state image sensing elements LGis arranged for every other column, and a ratio between the numbers ofsolid-state image sensing elements DR, solid-state image sensingelements DG, and solid-state image sensing elements DB is 1:2:1, and aratio between the numbers of solid-state image sensing elements LR,solid-state image sensing elements LG, and solid-state image sensingelements LB is 1:2:1.
 3. An image sensing apparatus including an imagesensor on which solid-state image sensing element which sense colorcomponents at a first sensitivity and solid-state image sensing elementswhich sense color components at a first sensitivity higher than thefirst sensitivity are alternately, two-dimensionally arranged,comprising: a first calculation unit that obtains a color component ofgiven pixel, sensed at the first sensitivity, by performinginterpolation calculation using color components of pixels each of whichis adjacent to the given pixel and is sensed at the first sensitivity,in a color image based on an image signal output from the image sensor;a second calculation unit that obtains a color component of a givenpixel, sensed at the second sensitivity, by performing interpolationcalculation using color components of pixels each of which is adjacentto the given pixel and is sensed at the second sensitivity, in the colorimage output from the image sensor, and a unit that outputs the colorimage in which color components of all pixels are determined by saidfirst calculation unit and said second calculation unit, wherein thesolid-state image sensing elements which sense the color components atthe first sensitivity include solid-state image sensing elements DRwhich sense R components at a first brightness, solid-state imagesensing elements DR which sense G components at the first brightness,and solid-state image sensing elements DB which sense B components atthe first brightness, wherein the solid-state image sensing elementswhich sense the color components at the second sensitivity includesolid-state image sensing elements LR which sense R components at asecond brightness higher than the first brightness, solid-state imagesensing elements LG which sense G components at the second brightness,and solid-state image sensing elements LB which sense B components atthe second brightness, and wherein said fist calculation unit obtains anR component, having the first brightness, of a pixel, sensed by thesolid-state image sensing element LR, by performing interpolationcalculation using an R component and a G component, both having thefirst brightness, of a pixel adjacent to the pixel sensed by thesolid-state image sensing element LR, and obtains a B component, havingthe first brightness, of the pixel, sensed by the solid-state imagesensing element LR, by performing interpolation calculation using a Bcomponent and a G component, both having the first brightness, of apixel adjacent to the pixel sensed by the solid-state image sensingelement LR, and said second calculation unit determines an R component,having the second brightness, of the pixel sensed by the solid-stateimage sensing element LR as a pixel value, in the color image, of thepixel sensed by the solid-state image sensing element DG, and obtains aB components, having the second brightness, of the pixel, sensed by thesolid-state image sensing element LR, by performing interpolationcalculation using a B component and a G component, both having thesecond brightness, of a pixel adjacent to the pixel sensed by thesolid-state image sensing element LR.
 4. An image sensing apparatusincluding an image sensor on which solid-state image sensing elementswhich sense color components at a first sensitivity and solid-stateimage sensing elements which sense color components at a secondsensitivity higher than the first sensitivity are alternately,two-dimensionally arranged, comprising: a first calculation unit thatobtains a color component of a given pixel, sensed at the firstsensitivity, by performing interpolation calculation using colorcomponents of pixels each of which is adjacent to the given pixel and issensed at the first sensitivity, in a color image based on an imagesignal output from the image sensor; a second calculation unit thatobtains a color component of a given pixel, sensed at the secondsensitivity, by performing interpolation calculation using colorcomponents of pixels each of which is adjacent to the given pixel and issensed at the second sensitivity, in the color image output from theimage sensor; and a unit that outputs the color image in which colorcomponents of all pixels are determined by said first calculation unitand said second calculation unit, wherein the solid-state image sensingelements which sense the color components at the first sensitivityinclude solid-state image sensing elements DR which sense R componentsat a first brightness, solid-state image sensing elements DG which senseG components at the first brightness, and solid-state image sensingelements DB which sense B components at the first brightness, whereinthe solid-state image sensing elements which sense the color componentsat the second sensitivity include solid-state image sensing elements LRwhich sense R components at a second brightness higher than the firstbrightness, solid-state image sensing elements LG which sense Gcomponents at the second brightness, and solid-state image sensingelements LB which sense B components at the second brightness, andwherein said first calculation unit obtains an R component, having thefirst brightness, of a pixel, sensed by the solid-state image sensingelement LB, by performing interpolation calculation using an R componentand a G component, both having the first brightness, of a pixel adjacentto the pixel sensed by the solid-state image sensing element LB, andobtains a B component, having the first brightness, of the pixel, sensedby the solid-state image sensing element LB, by performing interpolationcalculation using a B component and a G component, both having the firstbrightness, of a pixel adjacent to the pixel sensed by the solid-stateimage sensing element LB, and said second calculation unit obtains an Rcomponent, having the second brightness, of the pixel, sensed by thesolid-state image sensing element LB, by performing interpolationcalculation using an R component and a G component, both having thesecond brightness, of a pixel adjacent to the pixel sensed by thesolid-state image sensing element LB, and determines a B component,having the second brightness, of the pixel sensed by the solid-stateimage sensing element LB as a pixel value, in the color image, of thepixel sensed by the solid-state image sensing element LB.
 5. An imagesensing apparatus including an image sensor on which solid-state imagesensing elements which sense color components at a first sensitivity andsolid-state image sensing elements which sense color components at asecond sensitivity higher than the first sensitivity are alternately,two-dimensionally arranged, comprising: a first calculation unit thatobtains a color component of a given pixel, sensed at the firstsensitivity, by performing interpolation calculation using colorcomponents of pixels each of which is adjacent to the given pixel and issensed at the first sensitivity, in a color image based on an imagesignal output from the image sensor; a second calculation unit thatobtains a color component of a given pixel, sensed at the secondsensitivity, by performing interpolation calculation using colorcomponents of pixels each of which is adjacent to the given pixel and issensed at the second sensitivity, in the color image output from theimage sensor; and a unit that outputs the color image in which colorcomponents of all pixels are determined by said first calculation unitand said second calculation unit, wherein the solid-state image sensingelements which sense the color components at the first sensitivityinclude solid-state image sensing elements DR which sense R componentsat a first brightness, solid-state image sensing elements DG which senseG components at the first brightness, and solid-state image sensingelements DB which sense B components at the first brightness, whereinthe solid-state image sensing elements which sense the color componentsat the second sensitivity include solid-state image sensing elements LRwhich sense R components at a second brightness higher than the firstbrightness, solid-state image sensing elements LG which sense Gcomponents at the second brightness, and solid-state image sensingelements LB which sense B components at the second brightness, andwherein said first calculation unit obtains an R component, having thefirst brightness, of a pixel, sensed by the solid-state image sensingelement LG, by performing interpolation calculation using an R componentand a G component, both having the first brightness, of a pixel adjacentto the pixel sensed by the solid-state image sensing element LG, andobtains a B component, having the first brightness, of the pixel, sensedby the solid-state image sensing element LG, by performing interpolationcalculation using a B component and a G component, both having the firstbrightness, of a pixel adjacent to the pixel sensed by the solid-stateimage sensing element LG, and said second calculation unit obtains an Rcomponent, having the second brightness, of the pixel, sensed by thesolid-state image sensing element LG, by performing interpolationcalculation using an R component and a G component, both having thesecond brightness, of a pixel adjacent to the pixel sensed by thesolid-state image sensing element LG, and obtains a B component, havingthe second brightness, of the pixel, sensed by the solid-state imagesensing element LG, by performing interpolation calculation using a Bcomponent and a G component, both having the second brightness, of apixel adjacent to the pixel sensed by the solid-state image sensingelement LG.
 6. An image sensing apparatus including an image sensor onwhich solid-state image sensing elements which sense color components ata first sensitivity and solid-state image sensing elements which sensecolor components at a second sensitivity higher than the firstsensitivity are alternately, two-dimensionally arranged, comprising: afirst calculation unit that obtains a color component of a given pixel,sensed at the first sensitivity, by performing interpolation calculationusing color components of pixels each of which is adjacent to the givenpixel and is sensed at the first sensitivity, in a color image based onan image signal output from the image sensor; a second calculation unitthat obtains a color component of a given pixel, sensed at the secondsensitivity, by performing interpolation calculation using colorcomponents of pixels each of which is adjacent to the given pixel and issensed at the second sensitivity, in the color image output from theimage sensor; and a unit that outputs the color image in which colorcomponents of all pixels are determined by said first calculation unitand said second calculation unit, wherein the solid-state image sensingelements which sense the color components at the first sensitivityinclude solid-state image sensing elements DR which sense R componentsat a first brightness, solid-state image sensing elements DG which senseG components at the first brightness, and solid-state image sensingelements DB which sense B components at the first brightness, whereinthe solid-state image sensing elements which sense the color componentsat the second sensitivity include solid-state image sensing elements LRwhich sense R components at a second brightness higher than the firstbrightness, solid-state image sensing elements LG which sense Gcomponents at the second brightness, and solid-state image sensingelements LB which sense B components at the second brightness, andwherein said first calculation unit determines an R component, havingthe first brightness, of a pixel sensed by the solid-state image sensingelement DR as a pixel value, in the color image, of the pixel sensed bythe solid-state image sensing element DR, and obtains a B component,having the first brightness, of the pixel, sensed by the solid-stateimage sensing element DR, by performing interpolation calculation usinga B component and a G component, both having the first brightness, of apixel adjacent to the pixel sensed by the solid-state image sensingelement DR, and said second calculation unit obtains an R component,having the second brightness, of the pixel, sensed by the solid-stateimage sensing element DR, by performing interpolation calculation usingan R component and a G component, both having the second brightness, ofa pixel adjacent to the pixel sensed by the solid-state image sensingelement DR, and determines a B component, having the second brightness,of the pixel sensed by the solid-state image sensing element DR as apixel value, in the color image, of the pixel sensed by the solid-stateimage sensing element DR.
 7. An image sensing apparatus including animage sensor on which solid-state image sensing elements which sensecolor components at a first sensitivity and solid-state image sensingelements which sense color components at a second sensitivity higherthan the first sensitivity are alternately, two-dimensionally arranged,comprising: a first calculation unit that obtains a color component of agiven pixel, sensed at the first sensitivity, by performinginterpolation calculation using color components of pixels each of whichis adjacent to the given pixel and is sensed at the first sensitivity,in a color image based on an image signal output from the image sensor;a second calculation unit that obtains a color component of a givenpixel, sensed at the second sensitivity, by performing interpolationcalculation using color components of pixels each of which is adjacentto the given pixel and is sensed at the second sensitivity, in the colorimage output from the image sensor; and a unit that outputs the colorimage in which color components of all pixels are determined by saidfirst calculation unit and said second calculation unit, wherein thesolid-state image sensing elements which sense the color components atthe first sensitivity include solid-state image sensing elements DRwhich sense R components at a first brightness, solid-state imagesensing elements DG which sense G components at the first brightness,and solid-state image sensing elements DB which sense B components atthe first brightness, wherein the solid-state image sensing elementswhich sense the color components at the second sensitivity includesolid-state image sensing elements LR which sense R components at asecond brightness higher than the first brightness, solid-state imagesensing elements LG which sense G components at the second brightness,and solid-state image sensing elements LB which sense B components atthe second brightness, and wherein said first calculation unit obtainsan R component, having the first brightness, of a pixel, sensed by thesolid-state image sensing element DB, by performing interpolationcalculation using an R component and a G component, both having thefirst brightness, of a pixel adjacent to the pixel sensed by thesolid-state image sensing element DB, and determines a B component,having the first brightness, of the pixel sensed by the solid-stateimage sensing element DB as a pixel value, in the color image, of thepixel sensed by the solid-state image sensing element DB, and saidsecond calculation unit obtains an R component, having the secondbrightness, of the pixel, sensed by the solid-state image sensingelement DB, by performing interpolation calculation using an R componentand a G component, both having the second brightness, of a pixeladjacent to the pixel sensed by the solid-state image sensing elementDB, and obtains a B component, having the second brightness, of thepixel, sensed by the solid-state image sensing element DB, by performinginterpolation calculation using a B component and a G component, bothhaving the second brightness, of a pixel adjacent to the pixel sensed bythe solid-state image sensing element DB.